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Abstract 


In the past few years significant interest has developed in using artificial neural networks to 
model and control nonlinear dynamical systems (8]. While there exists many proposed schemes 
for accomplishing this and a wealth of supporting empirical results, most approaches to date tend 
to be ad hoc in nature and rely mainly on heuristic justifications. The purpose of this project 
was to further develop some analytical tools for representing nonlinear discrete-time input-output 
systems, which when applied to neural networks would give insight on architecture selection, prun- 
ing strategies, and learning algorithms. A long term goal is to determine in what sense, if any, 
a neural network can be used as a universal approximator for nonlinear input-output maps with 
memory (i.e., realized by a dynamical system). This property is well known for the case of static 
or memoryless input-output maps [1,5]. 

The general architecture under consideration in this project was a single-input, single-output 
recurrent feedforward network described by the n nonlinear difference equations 

x{t + 1) = <r(Ai(t) + 6u(0)» *(°) = *o C 1 ) 

y(0 = cx (t), ( 2 ) 

where < 7 , » = 1, . . . n are the individual activation functions for each neuron, 6 and e are the input 
and output connection vectors, A is the recurrent neural network weight matrix, and *(•) is the 
state vector composed of the outputs of each neuron [4]. For the purpose of modelling input-output 
behavior we found it convenient to introduce a state-space transformation z = a *(i) such that 
the system in (l)-(2) is input-output equivalent to 

z(H-l) = /(*(«)) + s(z(0M0. z(0) = <r- 1 (x o )iz„ (3) 

v(0 = M*(0)> (4) 

where /(*) = Aa{z), g(z) = b, and h{z) = c<r~ l (z). The main advantage to using this latter 
state-space model is that the system is affine in u, and thus, more amenable to analysis by the 
geometric methods used in nonlinear control theory [2,6] . The basic objective was then to determine 
a tractable functional expansion of the input-output map corresponding to system (3)-(4) in terms 
of the realization (f,g,h,z 0 ). A Volterra type expansion 

y(t) = T E Vi k (i„...,ia)tt(t-ii)...«(«-*a) ( 5 ) 

fc=0*|...u = l 
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haa been published in the literature [4,9], but the expressions for the Volterra kernels, V k , are too 
complex for any serious analysis. Thus, we took the alternative approach of deriving a discrete- 
time analogue of the Fliess functional expansion for general discrete-time nonlinear systems and 
then applying it in particular to the neural network realization in equations (3)-(4). The resulting 
expansion was significantly simpler in structure than those published to date and appears promising 
for future analysis of such systems. 

An interesting innovation resulting from this work was the discovery of a natural adaptation of 
Fliess’s work to discrete-time systems. This subject has been studied by other researchers [7,10], 
but it appears that this particularly tractable approach has been overlooked. One of two main 
results is described in the following theorem [3]: 

Theorem Suppose the sequence {t»(fc)} is bounded in magnitude on [0,7V] and each function of 
a discrete-time realization (f,g,h,z„) is analytic on an open subset U o/8? n . Then for sufficiently 
imaii TV, the input-output mapping can be represented by a convergent generating series 


F:u-»y= (6) 

ner 

where 

/* is the set of multiindices for the index set I — {0,1}; (7) 

e{n) = c{ik ■ ■ • *o) = L„. o . . . L 0 . k h(*„) (*,(*) = /(*) - *); (8) 

E n (t) = £,*..<«(*) = E«. fc (i)^ k -|...o0') [0,7V] (9) 

J — 0 

(B$(t) = 1, tio(0 = 1, and u t {t) = u(t))- (10) 

(L 9 h denotes the Lie derivative of h along g .) 


Clearly this result applies to the system (3)-(4). Furthermore, as in the continuous-time case, 
it is possible to derive a simple series expansion for each Volterra kernel in terms of the coefficients 
{ c(rj ) : r\ G /*}. The details of this analysis are reported in [3]. 
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